Location Based Services Quality Assessment

ABSTRACT

A technique to gather and assess the quality of a location based services provided to a mobile wireless device via a wireless network includes providing location-based service information by referring to a target location of a requested item of interest within a proximity of the present location of the mobile wireless device. Subjective quality or accuracy information can be submitted to the wireless network from the from the wireless device. The subjective quality or accuracy information is correlated with event and condition information affecting the wireless network. Reports can be generated which indicate the quality and accuracy information of the target location information. The reports are useful for troubleshooting location-based service inaccuracy.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part application of U.S. patentapplication Ser. No. 11/227,972, filed Sep. 15, 2005, entitled“Assessing Performance and Quality of a Mobile Communication Service”,which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The technical field is generally related to communication systems andmore specifically related to assessing the quality of a service providedvia a communications system.

BACKGROUND

Maintaining quality of service in the eyes of customers is an importantfactor in services industries. This is the case in the wireless/mobilecommunications industry, as well as the traditional wireline telephoneindustry where customers often base their decisions to join, or staywith a particular service provider based on the quality of the servicesprovided. For example, with respect to wireless telephones, events suchas recurring dropped calls, poor sound quality during calls andunexpected unavailability of service may drive customers to seek a newservice provider, especially given rising standards for call quality.

Currently, various techniques exist for monitoring quality of service inthe wireless communications industry. Some of these techniques, such asPESQ (Perceptual Evaluation of Speech Quality) obtain objective qualityof service information. For example, PESQ measures voice quality bycomparing an input test signal with the signal output across aconnection. Another objective technique, ITU-T E-model (e.g., ITU-T G.107) predicts conversational MOS (mean opinion score) from IP networksand/or terminal parameters. Subjective techniques also exist where testmobile devices are established to monitor sound quality. Serviceproviders often arrange for such testing immediately followingdeployment of new networks, new network elements (such as base stations)and network changes or upgrades, etc.

The techniques described above are often difficult to implement, may belimited in their capability to monitor an entire network, and may beexpensive, especially in the case of call quality monitoring techniquesthat utilize specialized infrastructure in the wireless environment.Many of these techniques may also cause unwanted load on the network. Inaddition, it is often not practical to implement such techniques on aregular basis. In addition, current quality monitoring techniques aredifficult to implement throughout the entire network, especially whennetworks may span large and diverse geographical area and the testdevices have to be frequently mobile to simulate as much as the realuser experience. For example, current quality monitoring techniques maybe implemented so that a single sector is used to report on the qualityof an entire region. Accordingly, it may be difficult to accuratelymonitor all areas of the network using such techniques.

SUMMARY

Described herein is a means for user feedback concerning observedaccuracy and quality of location-based services in a wireless networkedsystem. In one embodiment, a wireless network provides a capability toaccept user-provided quality feedback information concerning a targetlocation data provided by the network in response to a query forlocation information from the user. Feedback data provided by the usercan be correlated to other aspects of system operation such as thepreset location of the wireless receiving device, the call traffic ofthe system, and other relevant information. Reports may be generated toassist a system engineer or system administrator in troubleshooting andadjusting the wireless network to lessen or eliminate the targetlocation data provided by the wireless system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example of an environment in whichlocation based services quality assessment can be implemented.

FIG. 2 is a block diagram showing the mobile communication device ofFIG. 1 in one embodiment.

FIG. 3 is block diagram showing components of the mobile serviceprovider administration system of FIG. 1 in one embodiment.

FIGS. 4A and 4B are flow diagrams showing examples of high level callquality assessment routines in some embodiments.

FIG. 5 is a flow diagram showing an example of an automatic callbackroutine for reporting on call quality in one embodiment.

FIG. 6 is flow diagram showing an example of an optional callbackroutine for reporting on call quality in one embodiment.

FIG. 7 is flow diagram showing an example of routine at the mobiledevice of FIGS. 1 and 2 for reporting on call quality using aquestionnaire in one embodiment.

FIG. 8 is flow diagram showing an example of routine at the mobileservice provider system of FIGS. 1 and 3 for reporting on call qualityusing a questionnaire in one embodiment.

FIG. 9 is a display diagram showing an example of a user interface foracquiring call quality information from a user of a mobile device in oneembodiment.

FIG. 10 is a display diagram showing a second example of a userinterface for acquiring call quality information from a user of a mobiledevice in one embodiment.

FIG. 11 is a block diagram of an example location quality assessmentsystem.

In the drawings, the same reference numbers identify identical orsubstantially similar elements or acts. To facilitate the discussion ofany particular element or act, the most significant digit or digits in areference number refer to the figure number in which that element isfirst introduced (e-g., element 204 is first introduced and discussedwith respect to FIG. 2).

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

Location based services quality assessment will now be described withrespect to various embodiments. The following description providesspecific details for a thorough understanding of, and enablingdescription for, these embodiments. However, one skilled in the art willunderstand that location based services quality assessment may bepracticed without these details. In other instances, well-knownstructures and functions have not been shown or described in detail toavoid unnecessarily obscuring the description of the embodiments.

It is intended that the terminology used in the description presented beinterpreted in its broadest reasonable manner, even though it is beingused in conjunction with a detailed description of certain specificembodiments. Certain terms may even be emphasized below; however, anyterminology intended to be interpreted in any restricted manner will beovertly and specifically defined as such in this Detailed Descriptionsection.

I. REPRESENTATIVE SYSTEM

FIGS. 1-3 and the following discussion provide a brief, generaldescription of a suitable computing/network environment in which thecall quality monitoring can be implemented. Although not required,aspects of the call quality monitoring are described in the generalcontext of computer-executable instructions, such as routines executedby a general-purpose computer, e.g., a server computer, wireless device,or personal computer. Those skilled in the relevant art will appreciatethat location based services quality assessment can be practiced withother communications, data processing or computer system configurations,including Internet appliances, hand-held devices (including PDAs),wearable computers, all manner of cellular or mobile phones,multi-processor systems, microprocessor-based, ASIC-based, or PGA-basedsystems, programmable consumer electronics, set-top boxes, network PCs,minicomputers, mainframe computers, and the like. Indeed, the terms“computer,” “device,” and “component” are generally used broadly andinterchangeably, and refer to any of the above devices and systems, aswell as any data processor.

Aspects of the call quality monitoring can be embodied in a specialpurpose computer or data processor that is specifically programmed,configured, or constructed to perform one or more of thecomputer-executable instructions explained in detail herein. Aspects ofthe call quality monitoring can also be practiced in distributedcomputing environments where tasks or modules are performed by remoteprocessing devices, which are linked through a communications network,such as a LAN, WAN, or the Internet. In a distributed computingenvironment, program modules may be located in both local and remotememory storage devices (e.g., including memory associated with fieldprograms, gate arrays, EPROM memory, etc.).

Aspects of the call quality monitoring may be stored or distributed oncomputer-readable media, including magnetically or optically readablecomputer discs, hard-wired or preprogrammed in chips (e.g., EEPROMsemiconductor chips), nanotechnology memory, photonic memory,biological-based memory, or other data storage media, including but notlimited to any one or more of the following: magnetic, optical, elecrooptical, semiconductor, and supper conducting. Indeed, computerimplemented instructions, data structures, screen displays, and otherdata under aspects of location based services quality assessment may bedistributed over the Internet or over other networks (including wirelessnetworks), on a propagated signal on a propagation medium (e.g., anelectromagnetic wave(s), a sound wave, etc.) over a period of time, ormay be provided on any analog or digital network (packet switched,circuit switched, or other scheme). Those skilled in the relevant artwill recognize that portions of location based services qualityassessment can reside on a server computer, while corresponding portionsreside on a client computer such as a mobile or portable device, andthus, while certain hardware platforms are described herein, aspects oflocation based services quality assessment are equally applicable tonodes on a network.

Referring to FIG. 1, a system 100 on which the quality of serviceassessment system may be implemented is shown. The system of FIG. 1 isan example of a GPRS (general packet radio service) system based on GSM(global system for mobile communication). However, location basedservices quality assessment may be implemented on other types of systemsand technologies including a variety of cellular systems, UMTS, HPDSA, .. . , as well as other broadcast, multicast and point to pointcommunication systems that are any combination of simplex, half duplex,full duplex including terrestrial broadcast, cable, satellite broadcast,satellite cell, and any one or any combination of PAN, LAN, WAN, MAN,or/and other network(s) including Bluetooth, and 802.11 family. Thesystem 100 includes a mobile device 102 (e.g., mobile phone, PDA,wireless laptop, etc.) in communication with a base station 104. A basestation controller 106 in communication with a serving GPRS support node(SGSN) 108 and a gateway GPRS support node (GGSN) 110 together supportpacket switched transactions, which are handled separately from circuitswitched traffic that is supported by a mobile switching center (MSC)111. The MSC 111 also serves as an access point for the Public SwitchedTelephone Network.

The SGSN 108, GGSN 110, and MSG 111 interact with a home locationregister 112 (HLR). In some embodiments, the HLR 112 is the primarydatabase of permanent subscriber customer information for the serviceprovider's mobile network. In the context of activated devices, the HLR112 may contain pertinent user information, including addressinformation, account status, and preferences. In some embodiments, avisiting location register (VLR) 114 manages requests from out-of-areasubscribers who are out of the area covered by their home system.

In the illustrated embodiment, the system 100 includes componentsassociated with quality of service assessment including a mobile serviceprovider administration system 120. The mobile service provideradministration system 120 may include a quality monitoring system 122, acall quality database 124, a customer care system 126, and a billingsystem 128, described in more detail with respect to FIG. 3. Customersmay interact with the mobile service provider administration system 120and its various components via the mobile device 102 and a wirelessservices network 116, as well as through other means, such as theinternet 118.

FIG. 2 shows a block diagram of a typical mobile communication device200, such as a mobile handset. While a mobile phone is shown as themobile communication device in FIG. 1, those skilled in the relevant artwill appreciate that location based services quality assessment can bepracticed with other devices and configurations, including Internetappliances, hand-held devices, wearable computers, multi-processorsystems, microprocessor-based or programmable consumer electronics,set-top boxes, PDAs, portable laptop computers, and the like. The term“mobile device” is intended to include all such devices.

The mobile device 200 has one or more internal or external antennas 202for receiving and transmitting electromagnetic signals such as radiofrequency signals. A transceiver 204 is connected to the antenna(s) 202and typically provides modulation and demodulation of the transmittedand received signals, respectively. A processor unit 206 connected tothe transceiver 204 may comprise a signal processor, microprocessor,ASIC, or other control and processing logic circuitry. The processorunit 206 may perform signal coding, data processing, input outputprocessing, power control, and other functions necessary forimplementing a mobile communication device. A customer may provide inputto the processor unit 206 via a keypad 208, microphone 210, or displaytouchpad 212. In turn, the processor unit 206 may provide information tothe customer via the display touchpad 212 or a speaker 214.

The processor unit 206 may access information from, and storeinformation in, a nonremovable memory 216 or a removable memory 218. Thenon-removable memory 216 may consist of RAM, ROM, EPROM, a hard disk, orother memory storage technologies. The removable memory 218 may consistof Subscriber Identity Module (SIM) cards, which are well known in GSMcommunication systems, or other well-known memory storage technologies,such as “smart cards.” Various applications, including text messageapplications 220 and quality reporting applications 220, could beimplemented in either the removable memory 218 or the non-removablememory 216. For example, the applications may include a user interfaceapplication (e.g., a Java applet) that allows a user to rate a call orother transaction (e.g., by pushing a particular button). Theapplications may allow rating in real time (e.g., while a call is inprogress) or, alternatively, after call has been completed. In someembodiments, an application used to rate a call or other communicationmay be located on a device that is separate from the device used toplace the call itself (e.g., a smart device with GPS compatibility sothat the precise location of the user can be identified if the phoneitself does not have such capabilities).

In some embodiments, a device location component 224 allows the locationof the device to be known to the wireless service provider, so that thewireless service provider can use this information (or pass it along)for the purpose of assessing call quality factors.

Referring to FIG. 3, a more detailed view of the mobile service provideradministration system 120 of FIG. 1 is shown. The mobile serviceprovider administration system 120 may include a call quality monitoringsystem 122 (also shown in FIG. 1). Various components of the qualitymonitoring system 122 may include a quality reporting customer interface302 and a quality reporting processing and analysis component 304. Bothof these components may communicate with a call quality database 124(also shown in FIG. 1). For example, in some embodiments the qualityreporting customer interface 302 may provide a way for customers tointeract with the quality monitoring system 122 and provide informationthat can then be processed by the quality reporting processing andanalysis component 304.

Both the data collected from the customer using the quality reportingcustomer interface 302 and the data resulting from processing by thequality reporting processing and analysis component 304 may be stored inthe call quality database 124. In some embodiments the quality reportingcustomer interface 302 may be limited to a physical device interface (asopposed to possessing application-based user interface characteristics).For example, if the user's mobile device includes a comprehensiveinterface for providing quality reporting information, the qualityreporting customer interface 302 may simply be an interface to retrieveinformation from the device. In contrast, if the user's mobile devicedoes not include a comprehensive reporting interface, the qualityreporting customer interface 302 may include various features that allowusers to answer questions and/or provide statements about the quality oftheir calls. For example, in some embodiments, quality reportingcustomer interface 302 may include automated telephone questioningsystems, a web server and related applications for providing onlinequestionnaires, etc.

In some embodiments, the quality monitoring system 122 may also includeother quality control components 306. For example these other qualitycontrol components may use objective standards for providing qualitycontrol information. Examples include PESQ systems, systems employingtest mobile devices, etc. In some embodiments, the information collectedby the other quality control components 306 may be compared withinformation obtained and retrieved by the quality reporting customerinterface 302 and processed by the quality reporting processing andanalysis component 304.

As described with respect to FIG. 1, the mobile service provideradministration system 120 may also include a customer care system 126and a billing system 128, which may optionally interact with the qualitymonitoring system 122 to address customer concerns such as droppedcalls, credits to service plans, etc.

Unless described otherwise below, aspects of location based servicesquality assessment may be practiced with conventional systems. Thus, theconstruction and operation of the various blocks shown in FIGS. 1-3 maybe of conventional design, and need not be described in further detailherein to make and use location based services quality assessment,because such blocks will be understood by those skilled in the relevantart. One skilled in the relevant art can readily make any modificationsnecessary to the blocks in FIGS. 1-3 (or other embodiments or Figures)based on the detailed description provided herein.

II. SYSTEM FLOWS

Referring to FIG. 4A, a high level call quality assessment routine 400may provide a means for customers to report on call quality inaccordance with some embodiments. At block 401 a call is initiated,which may include initiation at the network level and/or at the devicelevel. At block 402 the call is in progress. At block 403 the call isterminated. For example, the user may have hung up the device to end thecall, or the call may have been dropped due to inadequate service.

At block 404 a quality reporting option is initialized upon terminationof the call. For example, the user may receive a message requestinginput on the quality of the call or may be given the option to make acall to an automated quality monitoring reporting system. Many optionsfor reporting are available without departing from the scope of locationbased services quality assessment. At block 405 the quality monitoringsystem collects call information. At block 406 the quality monitoringsystem performs exploratory and/or automated data analysis. As a resultof this data analysis, the quality monitoring system may identify issuesnot detected by objective assessment of individual network modes (block406 a); quantify actual customer experience across the entirecommunication network (block 406 b); assess the impact of changes madeto the communication network (block 406 c); identify, localize, andresolve call quality degradation issues (block 406 d), etc.

FIG. 4B shows a high level call quality assessment routine 420, similarto the routine 400 of FIG. 4A, but where the call quality information iscollected from the user while the call is still in progress. For exampleat block 421 a call is initiated, which may include initiation at thenetwork level and/or at the device level. At block 422 the call is inprogress. At block 423, the quality monitoring system collects callinformation during the call. For example, the user may be able to push abutton on the phone each time he or she feels that call quality duringthe call is below an acceptable level. At block 424 the call isterminated. For example, the user may have hung up the device to end thecall, or the call may have been dropped due to inadequate service. Atblock 425 the quality monitoring system performs exploratory and/orautomated data analysis based on the information collected at block 423.Some of this analysis may also occur while the call is in progress. As aresult of this data analysis, the quality monitoring system may identifyissues not detected by objective assessment of individual network modes,quantify actual customer experience across the entire communicationnetwork, assess the impact of changes made to the communication network,identify, localize, and resolve call quality degradation issues, etc.

FIG. 5 is a flow diagram showing an example of a routine 500 forcollecting quality information is collected by an automatic callbackfeature. At block 501 the call is terminated. At block 502 the qualitymonitoring system sends a message to the device to perform auto callbackupon completion of a call. Alternatively, the device may be programmedto automatically place such a call, depending on the conditions presentat the device (e-g., termination of a call by the user, termination of acall due to loss of service or other network problem, etc.

At block 503 the quality monitoring system receives a callback from themobile device. When this occurs, mobile device may ring as the call isbeing placed automatically by the mobile device. In some cases the usermay have the option of hanging up or discontinuing the feedback call ifthe user does not want to provide input at this time. At block 504 thequality monitoring system receives user input via the automatic call.The routine 500 ends after block 504.

FIG. 6 is a flow diagram showing an example of a callback option routine600 for providing an opportunity for the user to optionally providefeedback on a call that was recently made. In this routine 600, the userhas the option of whether or not to continue with a reporting callbefore the call is made. At block 601 a call is terminated. At block 602the quality monitoring system sends a message to the device to promptthe user for a callback. Alternatively, the device may be programmed todisplay such a message without a prompt from the quality monitoringsystem. For example the user may receive a message displayed on his orher device screen requesting whether he or she would like to call backand report on call quality at the current time.

At decision block 603, if the user selects to go forward with thecallback option, the routine 600 continues at block 604. Otherwise theroutine ends without reporting. At block 604 the routine receives anauto callback call from the device in response to the user's selectionto provide feedback. At block 605 the quality monitoring system receivesuser input. After block 605 the routine ends.

FIG. 7 is a flow diagram showing an example of a routine 700 for apost-call questionnaire, where the questionnaire is executed by aprogram on the mobile device itself. At block 701 the call isterminated. At block 702 the routine 700 begins executing at the mobiledevice. At block 703 the routine 700 poses a next question to the user.At decision block 704 if the routine 700 has posed a last question tothe user, the routine 700 continues at block 705. Otherwise, the routine700 loops back to block 703 to pose the next question to the user. Atblock 705, assuming all questions have been posed to the user, theroutine 700 sends a response to the call quality monitoring system. Forexample, the routine 700 may provide data comprising a completed seriesof questions or statements by the user on call quality.

FIG. 8 is a flow diagram showing an example of a routine 800 forpresenting a post call questionnaire to the user of a mobile device,where the routine 800 is executed from the quality monitoring system. Atblock 801 the call is terminated. At block 802 the routine 800 contactsthe device to initiate connection with the questionnaire application. Atblock 803 the routine 800 provides a next question to the device. Atblock 804 the routine 800 receives a response from the device (e.g.,provided by the user as a response to a question in the questionnaire).At decision block 805, if the last question has been posed, the routine800 ends. Otherwise, the routine 800 moves back to block 803 to providethe next question to the device.

III. USER INTERFACE FOR COLLECTIONS CALL QUALITY INFORMATION

FIG. 9 is a display diagram showing various examples of providing a userof a mobile device with an option to answer questions relating to callquality. With respect to display 902, a call has been dropped due to aservice problem, and the user is asked: “Would you like to report oncall quality now?” In this example, the user may select either “yes” or“no” using appropriate device keys.

In another example shown in display 904, a message provides: “Rate thequality of your last call now and receive free airtime minutes.” Theuser may then respond “yes” or “no” as to whether he or she wishes torate the call at the present time.

In a third example shown in display 906, a message provides three ratingoptions: “Select 1 to report on call quality now”; “Select 2 to reporton call quality later”; and “Select 3 to opt out of reporting callquality in relation to the current call.” If the user selects the firstoption, the device may present a text questionnaire to the user or makea call to an automated quality reporting system, as described in moredetail in the preceding flow diagrams and with respect to FIG. 10.Alternatively if the user selects to report on call quality later, theuser may be given the option to provide feedback at a later time via anyone of a number of means (e.g., Internet questionnaire, automated, call,text message on device screen, etc.).

FIG. 10 is a display diagram showing an example of a user interface forreporting call quality. Display 1002 shows a text mode where the user isasked to report on call quality after a dropped call using textmessaging or similar means. For example the user may be requested toanswer the question: “Did the signal strength vary during your call?” Tothis the user may respond either “yes” or “no” using appropriate buttonson his or her device.

Display 1004 illustrates reporting on call quality using an audio mode(e-g., automated telephone system). For example, the user may be askedto “describe the quality of your last call.” The user's response maythen be recorded and or automatically processed. In another example, theuser may be asked: “What was the signal strength just prior to the timethat the call was dropped? For poor select or say 1, or moderate . . . ”Another question that the user may be asked is “Have you had problemswith dropped calls in this area before?”

While specific examples are given here for the purpose of illustration,regardless of the mode used (e.g., text mode 1002, in an audio mode1004, or in another mode not illustrated here), any number of questionsmay be asked to the user for quality reporting. While by no means allinclusive, the table below shows examples of other types of questionsthat may be posed to users in determining call quality:

Sound Quality On a scale of 1-5, with 5 being the best quality sound,how would you rate the quality of the sound of your call? Were you ableto hear the person on the other end of the line clearly during the call?Never? Rarely? Most of the time? Always? Was the person on the other endof the line able to hear you during the call? Never? Rarely? Most of thetime? Always? Do you have repeated problems with sound quality whenplacing calls in this area? Noise On a scale of 0-5 with 0 being nodetectable background noise, how would your rate the amount ofbackground noise of your call? How often were you able to hear unwantedbackground noise during your call? Never? Rarely? Most of the time?Always? What was the severity of background noise during your call?Greatly affected ability to hear content of call. Somewhat affectedability to hear content of call. Did not affect the ability to hearcontent of call but was annoying. Was audible but not distracting. Nodetectable background noise present Do you have repeated problems withsound quality when placing calls in this area? Dropped Call Did thesignal strength vary during your call? What was the signal strength atthe time the call was placed? What was the signal strength just prior tothe time the call was dropped? Have you had problems with dropped callsin this area before? How many calls do you make in the area that thecall was dropped? How many times did you attempt to reconnect the samecall? Subjective Reponses Describe the quality of your last call.Describe your overall satisfaction with the quality of your last call.Describe how the quality of your last call could have been improved.Describe your overall experience with your wireless communicationservice provider. Describe your overall satisfaction wit your wirelesscommunication service provider. Other Did you have dial tone problemswhen making this call? Press 1 for yes. Press 2 for no. Rate the callset up time for this call. Please indicate if there were issuesdisconnecting from this call. Please indicate if there were problemshanding off between cells during this call.

In addition to the above, other factors and techniques may be used inrating calls. For example, various different types of rating scales maybe implemented. In another example, users may be provided withdescriptive icons or graphics to select from to make call rating fastand easy.

IV. LOCATION ACCURACY ASSESSMENT

In another aspect of location based services quality assessment,location service quality may be assessed from a user perspective. Inthis aspect, the user can provide feedback on the perceived quality oflocation services provided by a wireless telephone network that arereceived on a user wireless device. It is possible that the presentlocation of a user, as provided by her mobile device, is incorrect. Andthus, location services based on the present location will be in error.Errors may occur in a location based service in multiple ways. Forexample, if the location based service provides to a user a targetlocation of a nearby restaurant within easy commuting distance of theuser's mobile device but the information is in error, there can bemultiple causes for the error. A mobile device can include a traditionalmobile and cell phones and other devices including smart phones,multifunction devices, dedicated devices (e.g., music and video players)wireless PDAs and computers (including handheld, palm held, laptop,desktop, wearable, etc.) and other electronic content storage anddelivery devices and media.

One such cause for error in a location service that provides a targetlocation with respect to a present location of the user wireless deviceis that the data representing the present location of the target, in theabove case, a restaurant, can be in error. That is, the databasecontaining the location of the target restaurant could be in error inthat the restaurant is correlated with incorrect location data as readby the database having the restaurant location data stored thereon. Asecond source of error could be that the cellular-based locationdetermination equipment could have inaccurate location data resulting inan incorrect present location of the user equipment. An example of thistype of error could be that the cell-tower location information, storedin a database, is in error. If so, present location determination of acell phone or other mobile device would, become inaccurate if thelocation determination equipment used the erred cell tower location as abasis for triangulation location of the cellular phone or other mobiledevice. A third source of error for cellular-based location services isin the actual determination of the resent location using the informationprovided. For example, if the location based-services use timedifferential of arrival (TDOA), angle of arrival (AOA), or triangulationprinciples to determine a user mobile unit location, then thecalculation itself may be in error, which provides false presentlocation determination information. Another example of error is in theuse of assisted global positioning system (AGPS) information. If AGPS isused, the GPS spacecraft ephemeris is provided to a GPSreceiver/processor in the mobile device which receives the spacecraftidentifiers. The AGPS cell system provides ephemeris and other referenceinformation about the GPS satellites and ground position directly to themobile device via the cellular network so that the GPS receiverprocessor can more quickly calculate position information at the mobiledevice instead of waiting many minutes for ephemeris information to bedown-linked from the GPS spacecraft. Thus, if the ephemeris, referencealtitude, or reference position information supplied by the cellularsystem is in error, so will be the present location information derivedfrom that information.

The user of the mobile device that receives location-based services,such as the location of nearby target points of interest (restaurants,hotels, gas stations, etc.) generally knows if the target location givento the mobile device is in error. This becomes evident if the user takesthe mobile device to the suggested target location, and the target itemof interest is not at the expected target location. Thus, the user ofthe mobile device is in a unique position to first determine if thecellular-system-provided target location information is incorrect and,if so, provide feedback on the quality of the received target locationinformation.

In one aspect of location based services quality assessment, the mobiledevice user can provide user-feedback to validate the accuracy of targetlocation-based services in a cellular network. After utilizing alocation based service, such as a search for a nearby target point ofinterest using a mobile device, the user is provided user-equipmentbased means to comment upon the accuracy of the location-based service.Various implementations are envisioned. In one embodiment, the user canprovide feedback via the mobile device onto which the location-basedservice was provided. In another embodiment, the user can use separatemeans, such as the internet, to provide user feedback, or a combinationof both means. Other embodiments include the user answering questionsvia a network interface, sending and/or reply to an SMS messageindicating the quality of the location-based service, the user can makea packet switched and/or circuit switched call to an automated attendantto provide information pertaining to the quality of the location basedservice target location data or a combination thereof. The user'sresponse can be stored in a database or the like, for further analysisand comparison with other users' feedback.

In one aspect of location based services quality assessment, thelocation service quality assessment does not have to compare an absolutelocation accuracy with the location information provided by the mobiledevice. In general, correlating the mobile device user's observations ofacceptable target location accuracy is sufficient to determine if thelocation-based services offered by the telephone network provider hasbecome de-valued because of target location accuracy problems. Forexample, a measured inaccuracy of 30 feet may be of little consequenceto a user and a user may not wish to report such an error. Indeed, auser may not even consider this level of target location accuracy to bea problem. But a target location accuracy error equivalent to a cityblock length or greater is most likely of great significance to a user.The provision of present location (spatial) and time (temporal)information further increases the value of the collected target locationaccuracy information. For example, in the case of wireless services,knowing where and when poor or good location service was received canhelp correlate the user reported data with other infrastructure datasuch as outage or geographic coverage issues and reports from otherusers. Thus, the use of accurate time (day/hour/minute) and a locationservice quality assessment can enhance the subjective location accuracyassessment value.

In application, a subjective location service quality assessment systemmay be applied to a variety of system applications. Example systemsinclude conventional wired systems having a LAN or WAN connection, suchas a voice over IP system, a terrestrial telephone system, a point topoint system, and a theater system. Wireless applications for thelocation service quality assessment discussed herein may includeterrestrial cellular, general RF communications systems, and satellitesystems.

One such embodiment is in a wireless telephone network having wirelessuser equipment which can be a mobile telephone, a PDA, a laptopcomputer, or other wireless end-user equipment having location-basedservices capability. FIG. 11 depicts an example system 1100 useful tosubjectively assess location accuracy. In one typical scenario, a useroperating the user equipment 1106, requests a location-based service viathe network 1104. The location-based service is typically a serviceoffered via the network to subscribers. Location-based services includequeries for the location of items of interest nearby the presentlocation of the user equipment. For example, such requests can includequeries for the locations of banks, schools, churches, shops,businesses, gas stations, town centers, bazaars, flea markets,recreational areas, scenic areas, wireless centers (hot spots), and thelike. A typical wireless location-based service may accept a query froma wireless device user for a target location. The request is received bythe wireless network and routed to a location-based services server(s)1102. The query is answered by determining the location of the wirelessdevice producing the request. This is performed using locationdetermination equipment as part of the location based services server(s)1102 or may be determined using present location determination equipmentin the user's wireless device. Whatever means is used, the presentlocation information is provided to the location-based service and alocation of the target item of the request is sent to the user wirelessequipment 1106 for display 1106A.

In other embodiments, the network 1104 may be either a wireless network,such as a cellular telephone network or other type of digital servicesnetwork supporting such user equipment such as an e-mail devices, PDAs,and laptop computers. The network 1104 may also be a wired network suchas a point to point system, a LAN or a WAN. At the user equipment 1106,the location based service information is received and displayed 1106Ato the user. An example service would be to locate nearby gas stationsfor a traveler requesting such information. In this instance, theestimated present location of the mobile device is determined by thenetwork and its present location estimation equipment 1102. Based on theestimate of the user equipment by the location based service equipment,the locations of nearby gas stations can be provided to the userequipment and displayed 1106A.

In this scenario, if the target location of the nearby gas stations iscorrectly provided to the user equipment, then the user would simply goto the nearest fueling station, and re-fuel the vehicle to continue hertravels. However, if the location information of the nearby gas stationsis incorrect or inaccurate, the user has an option to report theinaccurate location information provided to her. As indicated above,correlation of a location inaccuracy with other information is useful toaid systems engineers and administrators in locating the cause for theerrant target location information.

The system 1100 permits a user to provide subjective feedback concerningthe location information displayed 1106A on the user equipment 1106. Thesubjective user input 1106B, also called subjective user feedback, maybe provided in a number of different ways. In one embodiment, a manualevaluation mode of subjective feedback may be used. In this mode, a userwould construct a message that is sent to the user data feedback datastore 1116. In the fully manual mode, a user may input the time, andconditions of the observed location inaccuracy as text entered throughthe keypad of the user equipment 1106. Another embodiment in manual modeis to record a voice message having the same information as the textmessage. The voice message would be sent to the user feedback store 1116for evaluation.

In another embodiment, the user equipment 1106 may solicit usersubjective information using an automated mode via the evaluation inputfunction 11106B. In the automatic mode, the user may select a menuoption while evaluating the location information. In this mode the localtime may be automatically entered using automated user equipment 1106resources such as an internal clock for time-tagging the feedbackinformation.

In one embodiment, where the user equipment 1106 is a wireless mobiledevice 200 as shown in FIG. 2, the quality reporting application 222provides the interface for the manual and automatic modes of thesubjective user input to the user equipment 1106. In general, theprocess of collecting and analyzing subjective user feedback comportswith the method of FIG. 4A. In the specific example of the acquisitionof user feedback during the time that location information is beingassessed by the user, the flow of FIG. 4B may be employed. In thespecific instance when an auto callback is used to collect the usersubjective information, the flow of FIG. 5 or 6 may be used. Post callquestions may also be employed to collect user information and the basicflow for that user interface may be conducted as per the flows of FIGS.7 and 8.

Returning to FIG. 11, in one embodiment, time information may beavailable as well as some local or present location information. Presentlocation information may take the form of a GPS receiver processor or anAGPS location mechanism. In accordance with the types of errors listedabove, even precise location information, when combined with faultydatabase information, can result in a poor quality location basedservice such as the location of a desired store, shop, park, and thelike. Thus, as an aspect of location based services quality assessment,the user equipment 1106 may also contain a time and/or location datasource 1106C as a resource for accurate time and location information.If implemented within the user equipment, the time function 1106C canprovide a time tag for subjective information input by the user. Forexample, in one scenario, where a user is evaluating search results fora restaurant, the user may, in real time, press menu keys on the userequipment to rate the quality of the location accuracy. The user mayinput a 5 for excellent quality and a 1 for very poor quality. As theuser actuates a soft key or other input device, a time function 1106Cpreferably located within the user equipment 1106 time tags theinformation so that the subjective rating information may be correlatedtemporally. As part of the quality evaluation process, the user may makemultiple requests for a location based service. After or during eachrequest, a quality assessment of the location service accuracy may bemade. In this manner, the wireless network support single or multiplequeries for a location-based service and the same wireless networksupports the provision of feedback on the service from a user wirelessdevice.

Considering the location function 1106C, it may reside within the userequipment or outside of the user equipment. Location determination maybe performed in a variety of ways. In one embodiment, if the locationfunction resides within the user device, a global positioning system(GPS) receiver/processor may be built into the user device such thatlocation information is provided and tagged to one or more usersubjective rating inputs. Differential GPS (DGPS) and Assisted GPS(AGPS) schemes may be implemented. Other satellite-based systems mayalso provide location information as well. In another embodiment withinthe user equipment, the location function may be such that the user isprompted to enter present location information manually onto the userequipment. The user-entered present location information, such as astreet name, for example, may then be sent out along with the subjectiveuser rating to assist in the location of the user device.

The location function 11106C may also be located outside of the userdevice and be part of the wireless network. An example of a locationfunction attached to the user network is a time difference of arrival(TDOA) calculation scheme. Another implementation is an angle of arrival(AOA) scheme. Those of skill in the art will recognize thatterrestrial-based locations mechanisms such as TDOA and AOA locationschemes can provide the location of the user equipment usingmeasurements from multiple base stations. This location information canbe used to augment the subjective rating input information from the userconcerning the quality of location based services received on the userequipment.

Subjective user feedback may be sent back through the network 1104 to bereceived and stored on a database 1116 on the network 1104 whichaccumulates the user feedback data for storage or buffering. The storage1116 may be local or remote storage component as part of a dataacquisition server, or may be implemented as an integral part of a basestation. The user feedback information is accessed from storage 1116 bya data correlator 1114. The data correlator can associate the userfeedback information with other network performance information toprovide useful event and condition information of the network 1104. Inone embodiment, the data correlator is part of the quality reporting,processing and analysis block 304 of the quality monitoring system 122shown in FIG. 3.

In on aspect of location based services quality assessment, networkperformance data is acquired by observing network events and conditions.In one example, a network monitor 1108 collects network performance in awireless network 1104. Network events such as call setups, teardowns,dropped packets and calls, switching failures, and outages are collectedusing the network performance monitor 1108. Also collected areperformance parameters and conditions such as bandwidth utilization,packet loss, call volume and call traffic handling, data link routingand switching element performance and switching configurations for basestation and cell tower configurations as well as capacity information.Such network events and conditions represent the performance, capacity,and configuration of the network 1104. The network performance andcapacity data is recorded using a storage device 1110.

The data correlator can input the user feedback information 1116 withthe network performance data from database 1110. By performing acorrelation between the subjective user feedback and the performance ofthe network, an association can be made whereby a cause and effect maybe identified. For example, if a network switch lost power or otherwisefailed during transmission of location information to a user, then theuser may experience a loss of signal or corruption of data at that time.By correlating the failure event to the subjective experience dataprovided by the user, a cause (the switch power failure) and the effect(flawed data) can be identified. In another instance, a change in thenetwork performance, such as when a marked increase in call volumeoccurred for a base station at the same time as a data request, and amis-read of location based data. Such a correlation can provide insightto a system administrator that a network traffic routing adjustment maybe needed for a group of base stations at a specific time of day.

Correlated information can be useful to determine the cause and effectof problems that cover a wide geographic area during by identifying acommon element such as latency on a server or switching or routingelement, problems that correlate to a specific area (e.g., a single cellsite sector) indicating a coverage, capacity or perhaps interferenceissue, problems that correlate to specific types of video content, orproblems that correlate to specific types of handsets or combinations ofhandsets. Thus, the value of correlating network performance data withsubjective video quality data becomes apparent.

In addition to the network performance data 1108 and the user feedback1116, the data correlator 1114 can access other information that isuseful for an assessment of location information displayed at the userdevice. In one embodiment, location based service information 1102server performance and capacity data is recorded on a storage device1112 which stores performance information concerning the location basedservices server. In one scenario, if the server has a fault, such as aninability to retain proper response time due to network faults or hightraffic demands, then the performance of the location based services1102 server could be adversely affected which would result in poor ordisrupted service at the user device 1106. Under such conditions,knowledge of the location based services 1102 server conditions would beuseful. The data correlator 1114 can also access storage device 1112 toobtain the location based services server performance and capacityinformation for correlation with the subjective information available instorage device 1116.

Another source of information that is useful to augment the userfeedback data and the network performance data is the road traffic andweather data available via database 1118. Here, as can be appreciated byone of skill in the art, weather aberrations can cause perturbations inperformance of user equipment 1106. Thus, data correlator 1114 canaccess the weather and road traffic data and associate the local weatherconditions between a relevant base station and the user equipment todetermine if the atmospheric conditions are affecting reception at theuser equipment. Road traffic information may like wise be useful becauseroad congestion may be an external source of wireless system usage. Forexample, given a congested section of roadway, cell phone usage may peakand aggregate to one or two cell towers or base stations. To accommodatethe above-normal call volume, bandwidth available for each call beingrouted through those base stations, and thus a location-based serviceperformance may be compromised. Knowledge of such road trafficconditions can thus support an analysis of degraded location service.

Another source of information useful to correlate with user feedbackdata is call data record information available at database 1120. Calldata record information contains information as to what communicationslinks were established, broken down, or dropped by the user equipment1106. When accessed, call data record information 1120 can be associatedby the data correlator 1114 in order to provide an event reference forthe location service quality rating information with respect to thespecific user equipment.

The data correlator 1114 can access information from the user feedbackdatabase 1116 and correlate the subjective information with informationfrom a variety of databases; the network performance database 1110, theserver performance database 1112, the weather and road traffic database1118, and the call data record database 1120 along with time andlocation information. All or any of this databases may be accessed bythe data correlator 1114. The data correlator can then associate thesubjective location service quality rating from the user with time andlocation information, network and location server performanceinformation, call records, and environmental condition data and identifyrelevant perturbations which can have a cause and effect upon thereceived location service quality presented by the user equipment to theuser. These results can then be reported to a system administrator viareport or in graphical display form 1122. The system administrator canthen take the results and determine if the location service quality datacan be improved by system hardware or software adjustments, or take someother action. Reports concerning the location-based service accuracy andquality assessment can be generated at 1102 to assist the administratoror system engineer in both troubleshooting and fixing unacceptablelocation-based service errors reported by wireless device users.

IV. CONCLUSION

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” Additionally, the words “herein,”“above,” “below” and words of similar import, when used in thisapplication, shall refer to this application as a whole and not to anyparticular portions of this application. When the claims use the word“or” in reference to a list of two or more items, that word covers allof the following interpretations of the word: any of the items in thelist, all of the items in the list, and any combination of the items inthe list.

The above detailed description of embodiments of location based servicesquality assessment is not intended to be exhaustive or to limit locationbased services quality assessment to the precise form disclosed above.While specific embodiments of, and examples for, location based servicesquality assessment are described above for illustrative purposes,various equivalent modifications are possible within the scope oflocation based services quality assessment, as those skilled in therelevant art will recognize. For example, while processes or blocks arepresented in a given order, alternative embodiments may perform routineshaving steps, or employ systems having blocks, in a different order, andsome processes or blocks may be deleted, moved, added, subdivided,combined, and/or modified. Each of these processes or blocks may beimplemented in a variety of different ways. Also, while processes orblocks are at times shown as being performed in series, these processesor blocks may instead be performed in parallel, or may be performed atdifferent times. Where the context permits, words in the above DetailedDescription using the singular or plural number may also include theplural or singular number, respectively.

The teachings of location based services quality assessment providedherein can be applied to other systems, not necessarily the systemdescribed herein. The elements and acts of the various embodimentsdescribed above can be combined to provide further embodiments. Aspectsof location based services quality assessment can be modified, ifnecessary, to employ the systems, functions, and concepts of the variousrelated technologies to provide yet further embodiments of locationbased services quality assessment.

These and other changes can be made to location based services qualityassessment in light of the above Detailed Description. While the abovedescription details certain embodiments of location based servicesquality assessment and describes the best mode contemplated, no matterhow detailed the above appears in text, location based services qualityassessment can be practiced in many ways. As noted above, particularterminology used when describing certain features or aspects of locationbased services quality assessment should not be taken to imply that theterminology is being redefined herein to be restricted to any specificcharacteristics, features, or aspects of location based services qualityassessment with which that terminology is associated. In general, theterms used in the following claims should not be construed to limitlocation based services quality assessment to the specific embodimentsdisclosed in the specification, unless the above Detailed Descriptionsection explicitly defines such terms. Accordingly, the actual scope oflocation based services quality assessment encompasses not only thedisclosed embodiments, but also all equivalent ways of practicing orimplementing location based services quality assessment under theclaims.

While certain aspects of location based services quality assessment arepresented below in certain claim forms, the inventors contemplate thevarious aspects of location based services quality assessment in anynumber of claim forms. For example, while only one aspect of locationbased services quality assessment is recited as embodied in acomputer-readable medium, other aspects may likewise be embodied in acomputer-readable medium. Accordingly, the inventors reserve the rightto add additional claims after filing the application to pursue suchadditional claim forms for other aspects of location based servicesquality assessment.

1. A method of assessing quality of a service provided via a wirelessnetwork, the service involving location information, the methodcomprising: establishing a communication to a wireless device, thecommunication comprising target location data with respect to a presentlocation of the wireless device; accepting subjective qualityinformation from the wireless device, the subjective informationgenerated by a user of the wireless device; correlating the subjectivedata with event and condition information affecting the wirelessnetwork; reporting the quality of the service in association with arelevant subset of the event and condition information, the quality ofthe service related to an accuracy of the target location data.
 2. Themethod of claim 1, further comprising the step of requesting, by thewireless device, information concerning the target location, before theestablishing step.
 3. The method of claim 1, wherein establishing acommunication to the wireless device comprises transmitting the targetlocation data of a place of interest to the wireless device in responseto a request for the target location issued from the wireless device. 4.The method of claim 1, wherein accepting subjective quality informationfrom the user device comprises: receiving data from the wireless deviceindicative of a subjective quality rating input by a user of thewireless device, the quality rating input provided after the userevaluated the target location data.
 5. The method of claim 4, whereinthe quality rating input is time-stamped by the wireless device.
 6. Themethod of claim 4, further comprising receiving multiple instances ofsubjective quality ratings by the user, each rating time-stamped by thewireless device.
 7. The method of claim 1, wherein correlating thesubjective data with event and condition information affecting thewireless network comprises associating the subjective data with at leastone of the group consisting of user call data records, road trafficrecords, weather records, location data, network performance data, andvideo server performance data.
 8. The method of claim 1, wherein thepresent location of the wireless device comprises location informationdetermined by one of the group consisting of global positioning systemdata, time differential of arrival data, and angle of arrival data, anduser input data.
 9. The method of claim 1, wherein reporting the qualityof the service in association with a relevant subset of the event andcondition information comprises reporting the quality of the service inassociation with data specifically related to network performance dataduring the time of the received subjective quality information.
 10. Asystem for assessing quality of a service that provides target locationinformation from a wireless telephone network, the system comprising: awireless device, connected to the wireless telephone network, thataccepts a subjective user input rating related to the service thatprovides the target location information; a data correlator receivingthe subjective user input rating and associating the input rating with acall data record of the user and at least one network event, theassociation producing a target location information quality assessmentrecord; a report generator, the report generator providing a pluralityof quality assessment records for examination, the records indicatingthe subjective user input rating providing an assessment of the qualityof the target location information.
 11. The system of claim 10, furthercomprising a network database connected to the data correlator, thenetwork database providing network data link routing and switchingelement performance data for association with the input rating.
 12. Thesystem of claim 10, further comprising a target location informationserver connected to the data correlator, the server providingperformance and capacity data for association with the input rating. 13.The system of claim 10, further comprising a road traffic and weatherdata source for association with the input rating.
 14. The system ofclaim 10, wherein the data correlator also associates time of the inputrating and a present location of the wireless device with a time of thenetwork event.
 15. The system of claim 14, wherein the time of the inputrating is determined from a time stamp provided by the wireless device.16. The system of claim 14, wherein the present location of the wirelessdevice is determined by one of the group consisting of globalpositioning system data, time difference of arrival data, angle ofarrival data, and a user input indicative of user location.
 17. Acomputer readable medium comprising computer instructions which, whenexecuted, perform a method to assesses a quality of received targetlocation information issued from a wireless telephone network, theinstructions causing execution of steps comprising: establishing acommunication to a wireless device, the communication comprising targetlocation data with respect to a present location of the wireless device;accepting subjective quality information from the wireless device, thesubjective information generated by a user of the wireless device;correlating the subjective data with event and condition informationaffecting the wireless telephone network; reporting the quality of theservice in association with a relevant subset of the event and conditioninformation.
 18. The computer-readable medium of claim 19, whereincorrelating the subjective data with event and condition informationaffecting the wireless telephone network comprises correlating thesubjective data with a time of the subjective data input, the presentlocation of the wireless device, and network switching performance data.